DOCUMENT RESUME 



ED 436 140 



IR 019 765 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Frick, Theodore; Monson, John A.; Xaver, Richard F.; Kilic, 
Gulsen; Conley, Aaron T. ; Wamey, Beatrice 
Navigational Structure on the World Wide Web: Usability 
Concerns, User Preferences, and "Browsing Behavior." 

1999-02 - 00 

8p . ; In: Proceedings of Selected Research and Development 
Papers Presented at the National Convention of the 
Association for Educational Communications and Technology 
[AECT] (21st, Houston, TX, February 10-14, 1999); see IR 019 
753 . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC0 1 Plus Postage. 

Comparative Analysis; *Design Preferences; Higher Education; 
Hypermedia; *Menu Driven Software; *Navigation (Information 
Systems) ; *Screen Design (Computers) ; User Needs 
(Information) ; Users (Information) ; *World Wide Web 
Browsing; Indiana University Bloomington; Usability; User 
Preferences; *Web Sites 



ABSTRACT 



There are several approaches a World Wide Web site designer 
considers in developing a menu structure. One consideration is the content of 
the menus (what choices are available to the user) . Another consideration is 
the physical layout of the menu structure. The physical layout of a menu may 
be described as being one of at least three different types. The traditional 
hierarchical disappearing menu structure is used most often. Persistent 
menus, or menus which remain on the screen, often accomplished by the use of 
HTML tables or frames, is a second approach. A third approach is the use of 
graphic or textual maps that show the path taken to a particular document. 

The purpose of this study was to determine whether visitors who are just 
browsing can do so more effectively under one of the three menu schemas 
described above. The comparative inquiry of the three treatments used 
volunteer graduate and undergraduate subjects (n=44) from Indiana University 
Bloomington. There were no statistically significant differences among the 
three groups with respect to subjects' conceptual understanding of the 
structure of the Web site. (Contains 27 references.) (MES) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



019765 " ED 436 140 



Navigational Structure on the World Wide Web: Usability Concerns, User 
Preferences, and “Browsing Behavior” 



EDUCAT.ONAI ggttSSEZ* 
n Th e y-j CENTER (ERIC) 

~ ced 

originating it. organization 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or 
document do not 
official OERI position or policy. 



document do no. fiSSrtttS in ,his 

official ncni u-.^ ' Y re Present 



Theodore Frick 
John A. Monson 
Richard F. Xaver 
Gulsen Kilic 
Aaron T. Conley 
and 

Beatrice Wamey 

School of Education, Indiana University 



Abstract 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 

-ZonoX- 






HhSOURCES 

INFORMATION CENTER (ERIC) 



A primary consideration in web site design is that the site be easily navigable for its intended audience. 
For "informational" sites ; where the primary purpose is to provide information about a service or 
program, a menu structure is often used to direct the users to different areas of the site. There are several 
approaches a site designer considers in developing a menu structure. One consideration is the content of 
the menus themselves (what choices are available to the user). Another consideration is the physical 
layout of the menu structure itself The physical layout of a menu may be described as being one of at least 
three different types. The " traditional , " hierarchical menu structure is used most often. Persistent menus ; or 
menus which remain on the screen , are often accomplished by the use of HTML tables or frames. A third 
approach is the use of graphic or textual "maps" which show the user the path taken to a particular 
document. The purpose of this study was to determine whether visitors who are "just browsing " can do so 
more effectively under one of the three menu schemas described above (hierarchical disappearing menus, 
persistent menus via frames or tables, and representations of navigational maps). 



Introduction 

A primary consideration in web site design is that the site be easily navigable for its intended audience. For 
informational sites, where the primary purpose is to provide information about a service or program, a menu 
structure is often used to direct the users to different areas of the site. 

There are several approaches a site designer considers in developing a menu structure. One consideration is 
the content of the menus themselves (what choices are available to the user). This is often accomplished through a 
combination of intuition and investigation. If one of the purposes of a site is to provide information that would 
normally be available through other means, such as direct inquiry to the information provider via a telephone call, 
consultation with those responsible for such inquiries can provide a well-established listing of frequently asked 
questions and answers. Another consideration is the physical layout of the menu structure itself. This consideration 
is often less dependent on inquiry or even intuition; it may be that aesthetics, visual design, and the limitations of 
servers and browsers determine how a menu is presented. The physical layout of a menu may be described as being 
one of at least three different types: 

1. The traditional, hierarchical menu structure is used most often. In this structure, menus are "pick lists" which move a 
user through a number of choices. At each step, the content of the previous menu disappears and is replaced by its 
subsequent menu. 

2. Persistent menus, or menus which remain on the screen, are often accomplished by the use of HTML tables or frames. 
Frames are actually sets of HTML pages which appear together. An advantage of this approach is that the menu can 
remain on the screen at all times, regardless of the content of the subsequent pages. In other words, the menu remains 
within one frame, and choices made via that menu are displayed within another frame. The disadvantage of this approach 
is that not all web browser clients can support frames; some usability experts actively discourage the use of frames because 
they tend to change the metaphor of the web page and can be confusing to some users. (Nielsen J., 1996). In an article by 
Andrews (1996) web designer, Frank Forbes, offered both frame and non-frame versions, his finding was that only 1 of 
the 10 users preferred the frames version 

3. A third approach is the use of graphic or textual "maps" which show the user the path taken to a particular document. An 
example of that type of design is: http://www.useit.com/alertbox/9612.htmL On that design type, the path one followed is 
shown as an active link at the top of each page. This may help the user to see where he is, how he reached there, and 
enable him to return to directly to the level he wants. 



In determining which approach is most useful and usable for visitors to a web site, designers often employ 
development and test procedure which can spot strengths and weaknesses of an emerging system. One process for 
doing this is described in Practical Web Development: A Systematic Process (Boling & Frick, 1998). 
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Needs analysis determines the typical questions users desire answers for and guides the development of the 
structure, layout, and content. Paper prototypes are developed for the proposed site. The paper prototype is tested 
with "typical” users. These users are given tasks, as identified in the needs analysis, and asked to complete these 
tasks. Revisions are made as necessary, and the paper prototype is converted to a computer-based prototype. Again, 
usability testing is conducted, this time with the computer prototype. Once again, the usability testing is conducted 
with users being asked to complete predetermined tasks. 

However, not all visitors to a web site arrive with particular questions in mind. These users are "browsing," 
or in other words simply visiting the site to see what it has to offer. They may return later with particular questions, 
but their first visit can be characterized as "just looking." Usability testing with predetermined tasks can provide 
valuable clues as to whether the site meets the needs of the information provider (is the site’s content logically and 
intuitively arranged, so that "typical" users' questions are answered?). However, predetermined tasks do not address 
the usability issues posed by visitors who are "just browsing." 

The purpose of this study is to determine whether visitors who are "just browsing" can do so more 
effectively under one of the three menu schemas described above (hierarchical disappearing menus, persistent menus 
via frames or tables, and representations of navigational maps). An additional research question pertains to the 
usability-testing procedure itself: should usability testing include unstructured tasks, as well as predetermined tasks? 
In this study, the content is three versions of the top-level structure of the web site for the Indiana University School 
of Education. 

This study examined which of the three menu designs worked better for a user who is not given a particular 
task. For example, after using a site for 10 minutes without a specific task, what would be the user’s conception of 
the structure of the web site? 

Literature Review 

Boling & Frick (1998) describe an approach to web design which depends largely on the determination of 
frequently-asked questions and designing usability tasks related to those questions. This approach is built upon the 
fundamental premise that users are looking for particular information and have predetermined goals when accessing 
a web site. This method for web design has proven successful in developing information- intensive sites at Indiana 
University. One shortcoming of this approach is its inability to incorporate what some have called “browsing” type 
behavior. “A major source of hypertext usability problems stem from the fact that hypertext systems are designed to 
encourage and facilitate browsing” (Smith, 1997). 

Implications of Browsing Behavior 

Browsing, due to its serendipitous nature, is difficult to simulate. When truly browsing a site, users have 
their own goal set, which may or may not be obvious to the researcher and which may change over time as 
information is gathered and refined. This may help to explain the findings from Remede, Gomez and Landauer 
(1987) which suggest that the main cause of inability to find information in a system is that users describe what they 
are looking for in different terms to those the system knows. 

Because the web is a relatively recent development, most research studies involving the web have been 
studies provide demographic information about users with only scanty information about how they are using it 
(Catledge, 1995). Recent studies by Raman (1996) about browsing behavior in particular, found that individuals 
enjoyed the feeling of exercising control over what they wanted to see, and the ability to tap into vast amounts of 
information on diverse topics. In evaluating a hypermedia library information system, Zhu (1995) found that users 
chose browsing much more often than querying and used content-based browsing much more often than using an 
index. 

Hendry and Harper (1997) discuss how opportunistic searchers (those who work the way they want to, as 
opposed to following a determined set of goals or strategies) might benefit from "simple systems that create a 
flexible and revealing dialogue between the system and user." They suggest that by representing search plans or 
navigational routes on a page, opportunistic searchers may be better equipped to perform informal problem-solving. 
However, usability tests which rely on predetermined tasks do not support the opportunistic searcher; those needs 
may be less-formally defined. 

Attention to cognitive organization of the user was also the focus of study in Coll (et al. 1993). This study 
found relevance in the common design guideline of using nine or fewer menu items, in addition to finding that good 
conceptual layout of the menu on screen is more important than good physical layout. 

The non-linear way in which we learn, and this similarity with the structure of hypertext was tested by 
Jannasch-Pennell (1996). This study investigated the impact of learner control on performance in a hypertext 
instructional environment. The study examined five learner levels across different educational levels (middle school, 
high school, undergraduate, graduate). Performance was measured through scoring on a criterion measure. Using 
covariance analysis, the study examined the performance of the educational levels against the variables of computer 



confidence/competence, level of statistical knowledge, experience with the Web, and gender. While the abstract 
identified the hypothesis of this study to be "the educational level and the interface characteristics have a direct 
impact on user performance," no mention of the study’s outcome was provided. 

Catledge (1995) points out that most usability studies that have been conducted are generally performed in a 
closed, single-authored system. It is apparent that given the complexity, size and ever changing nature of the web, 
any global scale usability study would prove less than fruitful. Because of the known propensity of many users to 
browse, additional testing measures should be taken to simulate such behavior, to ensure more authentic 
generalizations from the research findings. 

Consistency and Persistence in Navigation 

Although pre-web” in its origin, a study by Lucas (1991) provides several recommendations of design for 
computer-learner interface, mostly based on previous research and testing. He suggests consistency in screen layout, 
using visual/graphical images to enhance textual descriptions, and menu selection interfaces that prevent the need for 
the learner to learn commands. Consistency is also a primary concern in the Apple Web Design Guide (1997), 
which states that ‘‘providing navigation links in standard places helps people to develop a perception of stability in 
your site and makes navigation easier.” Among his 16 recommendations on web page design Tillman (1997) makes 
these navigation-related prescriptions: each node/destination be able to stand alone, provide visual clues indicating 
users’ selections are being processed, and employ labeled and unlabeled links. 

Hardman (1989) used observational evidence to decipher that many users return frequently to the screen 
from which they began, which may suggest that users desire to get back to a known place instead of working out a 
route which will take them from where they are to where they want to go. Additional credibility is given to this 
finding through a study at the Georgia Institute of Technology involving 107 users over a three week period in which 
using the browser’s “Back” command or button accounted for a full 41% of all navigation. 

Catledge’ s study at GIT revealed that the typical user accessed on average 10 pages per server, which 
would lead to the conclusion that the “must see” information should be accessible within two or three jumps of the 
initial home page (two/three navigation’s in, two/three out, performed two/three times) a finding also suggested by 
Apple. 

Other relevant studies in persistent navigation include, Smith (1994) where testing was done using both 
two-and three-dimensional, spatial and schematic navigational aids, as well as Andrews (1996) where the use of a 
two-dimensional structure map which visualizes link and membership relationships, with a three-dimensional 
landscape visualization, presenting a combined display of both hierarchical structure and hyperlink relationships 
helped to prevent users from losing their orientation. 



Utilizing Mental Maps in Information Recall 

In her doctoral dissertation, Tyan (1989) showed that a top level structure awareness greatly enhanced 
readers recall performance. This study was done in part by having readers complete a schematic diagram of a 
passage previously read, and found a positive correlation between knowledge of structure and recall of information 
within the structure. Although this study was done with print-based text, the principles of recall may apply to web 
based text as well. 

Increasingly, researchers and design practitioners are calling for the integration of spatial metaphors when 
designing Web pages. Dieberger and Bolter (1995) argue the Web is inherently geographical, leading users to 
organize their search strategies in spatial forms. And Heid (1996) suggests image maps actually help improve 
navigation by allowing developers to include backward compatibility with forward-looking design. 

In one study, Philleo asked a group of preservice teachers, for whom the Internet was a relative ly-new area 
of study, to draw "...a diagram or picture of the Internet as you understand it” (1995). Philleo concluded that "it may 
be that in order to solve problems that arise with Internet use or to be fully empowered to explore its resources, some 
sort of realistic-looking mental image might prove helpful.” Taking Philleo’s findings further - or perhaps, by scaling 
it down to the scope of a single web site - it may be that a browsing user's pictorial image may provide information 
about what the user thinks the site contains (see also Apple, 1997). Beyond this, by providing a persistent site map, 
one which shows how a user arrived at a particular page, a user may be helped in maintaining a mental image of the 
site, making subsequent visits more fruitful and productive. 

In a similar study to this proposal, Chang (1995) used three types of hypertext menu designs while testing 
against individual difference variables. The three menu styles were 1) hierarchical structure signaled by explicit 
menus and referential links presented by embedded menus, 2) hierarchical structure signaled by embedded menus 
and referential links presented by embedded menus, and 3) referential links presented by embedded menus without 
structural information. The two individual variables were cognitive style (field-dependence versus field- 
independence) and epistemic beliefs and preferences (simple versus complex). The study population of 52 students 
at the University of Illinois yielded the following conclusions: Cognitive style was found to an influential factor on 
searching performance as field-independent users fared far better, especially when no structural information was 
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provided; Users' beliefs about whether knowledge is simple or complex did not affect searching performance; 

Program #1 proved to be the most efficient, but did not always produce the best search scores; and most novice 
subjects given simple search tasks chose linear links over non-linear hypertext. 

In summary, the literature indicates that: 

1 . Current research shows that people use various types of browsing behavior in navigating through web sites, and such 
behavior is difficult to simulate in research activities. 

2. Consistency in screen layout and design assists users in navigation through a particular web site. 

3. Users tend to return frequently to the “top level*’ or screen from which they first entered a site rather than working out 
paths which will take them where they want to go. 

4. A mental awareness of the structure that information is presented in aides in the recall of the material. 

Methodology 

The comparative inquiry of the three treatments used volunteer graduate and undergraduate subjects from 
Indiana University Bloomington. As subjects were recruited, they were assigned at random to one of the three 
treatments (total N = 44). In keeping with University policy concerning the use of human subjects, participants were 
given a copy of an Informed Consent document, which explained the nature of the study and assured anonymity. 

After reviewing and signing the Informed Consent document, the subjects were asked to complete a brief 
questionnaire to determine the their level of familiarity with the current web site. 

Because many of the hypertext links within this site were originally designed to take the user to other web 
sites, and because our research questions concerned only the navigability of the site in question, off-site links were 
changed so that they would produce an error message (normally, this would produce a "404 not found error 
message; the server was reconfigured to deliver a more user-friendly message). The error message was pre-set to 
describe the reason the document was missing, and the subjects were told that finding one of these pages was a 
possibility; that if they did, they should use the browser’s back button to return to the previous page and continue 
browsing. 

The subjects were then shown the home page for the treatment they were assigned to; they were told that 
their assignment was to simply browse the site for a period of time and get an idea of its content. The three 
treatment conditions can be viewed on the Web at URL: http://education.indiana.edu/~frick/aect99/designs.html (A = 
non-persistent (changing) hierarchical menus displayed as vertical lists of choices with non-hot exemplars to the 
right; B = A + persistent top-level menu displayed as a vertical list of choices in the left column of the screen; C = A 
+ horizontal hierarchical lists of choices at the top of the screen that represent the path taken down the hierarchy 
[textual map] ). 

The subjects then proceeded to browse the site while the researchers observed and recorded; the starting 
time, number of "back button" clicks, and subject comments were the primary observations made. Browser back 
button clicks were recorded as having been "forced" or "unforced." Forced back button clicks were the result of 
having arrived at a "document not found" page (as described above); unforced back button clicks were all others. 

After browsing the web site for about ten minutes, the researcher asked the subject to stop. A questionnaire 
was then completed which gathered general information about the subject and their opinions about certain aspects of 
the site. General information included native language, major, and various aspects of computer use; opinions about 
the web site concerned navigability, affective response, and understanding of the purpose and navigational structure 
of the web site. The subjects were also asked four open-ended questions regarding what they liked or did not like 
about the web site. 

In the final aspect of the session, the subjects were asked to complete a conceptual map of the web site. 
Because recall of the material may be a factor in the subjects' conceptual understanding of the site, it was decided 
that at least ten minutes should elapse between the end of the browsing activity and the beginning of the conceptual 
map test; if the subject completed the questionnaire in less time, an unrelated activity was begun to take up the 
remainder of the time. 

Analysis 

Following the data collection process described in the methodology, each subject's responses were reviewed 
and analyzed. The data yielded through the testing included (1) overall browsing time, (2) total unique pages visited, 
(3) conceptual map score, (4) the number of times the "back" button was used, and (5) ancillary data such as 
comments made during testing and written responses on the post-test questionnaire. The web page logs were also 
retrieved for each subject's testing period to verify the team member's observations. 

Conceptual map tests were first scored independently by two persons, then the average of two was 
calculated. However, we found the correlation between two persons' scores was not sufficient (0.74). In order to 
improve the reliability of scoring, two scorers scored each conceptual map together by reaching a consensus score 
for each. Then, three tests from each design were chosen randomly and a blind observer scored them without 
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knowing which design they were. The blind observers’ scores were then correlated with the consensus scores, but 
again the correlation was low (0.68). 

In the statistical analysis, the recorded variables were coded into an SPSS file and subjected to a number of 
statistical manipulations. In particular, an attempt to correlate total browsing time and conceptual map scores, as 
well as total unique pages visited and total browsing time was made through correlation (Pearson r) and a one-way 
analysis of variance (ANOVA). 

Unfortunately, the data collected on back button usage was not usable. This occurred for a number of 
reasons. Observers attempted to record usage of the back button as an "ordinary” choice of the user compared to a 
forced choice. Since all pages in the designs that led to external links under normal conditions were closed, users 
were forced back to a previous page. Due to an oversight on the research procedures, not all observers distinguished 
between these two uses of the back button. Furthermore, the web logs could not accurately verify the types of back 
button usage as had been anticipated, so the data was discarded. 

Research Findings 

An ANOVA was undertaken to see if there were any statistically significant differences among three 
different designs. Average conceptual map scores in the three designs were nearly identical. 



Table 1. Means and Standard Deviations for Conceptual Map Scores in the Three Menu Designs 



Design 


Mean 


Standard Deviation 


A (traditional non-persistent hierarchical menu) 


n=14 


3.43 


2.62 


B (persistent main menu - left side) 


n=15 


3.87 


1.92 


C (persistent textual map — top left-to-right) 


n=15 


3.27 


2.31 



There were no statistically significant differences among the three groups ( p < .05) with respect to subjects’ 
conceptual understanding of the overall structure of the Web site. Based on both the statistical analysis and the 
interpretation of test subject’s written and verbal comments, few subjects noticed the differences between the three 
test designs. Observers noted that subjects tended to use the non-persistent links for navigation that were present in 
all three designs. Most subjects did not use the persistent menu on the left (sidebar) or the textual chain (top) for 
navigation while browsing, since they apparently did not "see” them. Moreover, these conceptual map scores were 
quite low with little variation among subjects within groups. Not only did this limit reliability of the measure and 
hence the possibility of significant differences, but it also indicated poor conceptual understanding of the Web site 
(lack of an accurate mental model). 

Pearson product-moment correlation’s were run between demographic and other variables and conceptual 
map scores. While most of the correlation values were not significant, there were statistically significant 
correlation’s between conceptual map scores and how often subjects used web the web per week (0.319) and the 
number of software products used frequently (0.338). These findings indicate those subjects who used the Web and 
computer software products more frequently were better able to grasp the conceptual structure of a new Web site in a 
relatively short period of time. 

Conclusion and Reflections 

While this study did not discover a design format that convincingly facilitates browsing behavior over other 
formats, as the study had intended, there are several possible interpretations, which will require further research 
before any firm conclusions can be reached. One interpretation is that subjects tend not to form mental models of 
Web site structures, which would explain the low conceptual map scores overall and lack of differences among the 
three menu formats. Spool et al. (1997) conclude from their experiences in usability testing that most people do not 
form mental models of Web sites. Another interpretation is that there are no differences in conceptual understanding 
that can be attributed to the three types of menu designs that were studied. Evidence to support this are observations 
that most users did not use those persistent navigation features because they did not notice them. If so, then we 
would not expect differences among the designs, since there were none from the user’s perspective What does 
appear to be supported by findings in this study is that more experienced computer users do seem to achieve a 
greater conceptual grasp of a Web site’s structure when browsing over a relatively short period of time. The number 
of software products used frequently indicates greater computer experience, as well as more frequent use of the Web. 

A number of points should be considered on how this study was conducted and how a similar future study 
might do things differently. As previously mentioned, tracking usage of the back button proved difficult in this 
study. In general, the speed at which subjects browsed through the pages indicates that a more efficient method of 
accurately following browsers' paths needs to be devised. While Web logs are useful to verify the pages a subject 
visits, they do not reveal if she or he uses the back button, or specially designed navigation buttons like the ones 
included in designs B and C as alternative shortcuts to previously visited pages. Moreover, browsers cache Web 
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pages recently visited, and no Web server log entries are made when they go back to those locally cached pages. 
Videotaping subjects’ use of the Web site (or capturing with a mediator and recording on video) would allow 
observers afterwards to more accurately track rapid navigation by users and the means by which they navigate. 

Future studies should also consider the type of test subjects employed. This study was limited by its use of 
mostly undergraduate students, many of whom were only first or second year students. Since the awarding of extra- 
credit was necessary to leverage an adequate supply of test subjects, the motivation of a large proportion of the test 
population, and thus the results of their test sessions could be questioned. Perhaps future studies should consider 
using a different population, such as professional staff of a school or department. This would also allow extending 
the testing period over a longer period, rather than just the average 10 to 30 minutes experienced with the student 
population. Indeed, an expanded test period of, for example, two days a week for 30 minutes over the course of 
several weeks would most likely yield more conclusive results than those found here. 

The last observation for potential future studies relates to measuring a users structural understanding of a 
web site. This study measured comprehension through the completion of a conceptual map. As cited in the literature 
review, there is evidence that users form their understanding of web page arrangements in spatial patterns. While the 
methodology used here was in accordance with previous studies, future studies may consider using an alternative 
measurement tool. 

In the case of this study, a more effective alternative method may have been to ask the user, after browsing 
the site for a period of time, to list what kind of answers this site could provide to them. While this approach would 
place greater responsibility on the researcher to interpret user's responses, it would eliminate the constrictions users 
faced in this study in terms of the way in which they were required to recall multiple first- and second-level headings. 
As noted earlier, the speed at which many of the subjects moved through the test sites may have inhibited their ability 
to form a spatial interpretation of the site. While the speed of navigation witnessed in this study suggests the designs 
presented the information efficiently, this efficiency had an unintended negative impact on comprehension and recall. 

One final thought on this study addresses the very nature of web browsing and the inherent difficulty in 
measuring it. Based on the observation of the test subjects, a question emerged related to the way the information in 
the design sites was presented. This question, which also applies to other web sites, is whether the manner in which 
information is presented creates self-forming goals within the user. Through observations made in this study, many 
test subjects "found” a topic or point of interest, which prompted them to continue searching for additional 
information related to their discovery. For example, when coming across the faculty and staff directory, the subjects 
began searching for one particular individual, usually a professor whose class they were enrolled in. This led them 
search that individual’s personal page, or look for their office in the School of Education building map. When these 
subjects ’’found” something of interest and began pursuing addition related information, they were no longer 
browsing, but they were now driven by a task. 
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